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Abstract 

The objective of this work is to validate math- 
ematically derived clock synchronization theories 
and their associated algorithms through experiment. 
Two theories are considered, the Interactive Conver- 
gence Clock Synchronization Algorithm and the Mid- 
point Algorithm. Special clock circuitry was designed 
and built so that several operating conditions and 
failure modes (including malicious failures) could be 
tested. Both theories are shown to predict conser- 
vative upper bounds (i.e., measured values of clock 
skew were always less than the theory prediction). 
Insight gained during experimentation led to alterna- 
tive derivations of the theories. These new theories 
accurately predict the behavior of the clock system. 
It is found that a 100-percent penalty is paid to tol- 
erate worst-case failures. It is also shown that under 
optimal conditions (with minimum error and no fail- 
ures) the clock skew can be as much as three clock 
ticks. Clock skew grows to six clock ticks when fail- 
ures are present. Finally, it is concluded that one 
cannot rely solely on test procedures or theoretical 
analysis to predict worst-case conditions. 

Introduction 

Many theories of clock synchronization have been 
proposed and subjected to the rigors of mathematical 
proof of correctness (see refs. 1 and 2). Few of these 
theories are validated by experiment . One of the dif- 
ficulties in validating clock synchronization theory is 
that the theory often predicts the behavior of the syn- 
chronization algorithm under failure conditions that 
are hard to replicate in the lab (e.g., the presence of 
a “malicious liar,” ref. 3). The objective of this work 
is to select a theory for validation, build a synchro- 
nization subsystem that is based on this theory, and 
subject this subsystem to a series of tests designed 
to validate the theory. 

The Interactive Convergence Clock Synchroniza- 
tion Algorithm (ICCSA) of Lamport and Melliar- 
Smith (ref. 4) was chosen as a test subject because 
of its use on the SIFT (Software Implemented Fault- 
Tolerance) computer (ref. 5) and the fact that the al- 
gorithm and the accompanying bounding theory had 
been recently subjected to the rigors of a mechani- 
cal proof (ref. 6). During the process of testing, it 
was found that the theoretical bound on the clock 
skew was larger than the observed maximum clock 
skew. Although the theory only guarantees an upper 
bound, this discrepancy led to inquiries into why the 
theory was not more accurate. In the course of this 
investigation, an alternative method for the deriva- 
tion of the expression for the clock skew bound was 
developed. This new expression accurately predicts 


the observed clock skew for the Interactive Conver- 
gence Clock Synchronization Algorithm. 

Lundelius has derived a clock skew bound (ref. 7) 
for the Midpoint Algorithm proposed by Dolev 
(ref. 8). The Dolev algorithm was programmed into 
the clock synchronization subsystem and tested. As 
with the ICCSA theory, the predicted bound was 
found to be greater than the observed clock skew 
(although only in extreme cases). With the insight 
gained from the previous derivation and applying a 
fresh approach to the worst-case analysis of the Mid- 
point Algorithm, a new expression is derived that 
accurately predicts the observed clock skew. 

In the following sections, expressions for the clock 
skew bound for both the ICCSA and the Midpoint 
Algorithm will be derived. A test plan will be 
introduced, and the design of the clock subsystem 
described. Results of the testing are presented and 
case studies are done. Finally, conclusions concerning 
this work are drawn. 

Symbols 


EH DM 

extended hierarchical design 
methodology 

fc 

clock counter frequency 

fr 

clock reference frequency 

HDM 

hierarchical design 
methodology 

ICCSA 

Interactive Convergence Clock 
Synchronization Algorithm 

m 

number of faulty clocks in a 
synchronizing set 

n 

number of clocks in a synchro- 
nizing set 

p,q,r,s 

processor designations 

R 

minimum length of synchro- 
nization period 

S 

minimum length of synchro- 
nization process 

T 

clock time 

T c 

time of clock correction 

T qp 

clock reading of processor p 
upon receipt of synchroniza- 
tion signal from processor q 

Ts 

time of synchronization signal 

t 

real time, (1 — p)T + e + to 

t* 

uncorrected clock function (see 
fig. 4) 



1 0 

real-time offset at T = 0 

V 

drift rate setting in clock 
subsystem peripheral 

A 

limit of perceived skew allowed 
in ICCSA 

A 9P 

perceived skew of processor p 
with respect to processor q 

6 

maximum skew between good 
clocks in a synchronizing set 

6 qp 

real-time skew between 
clocks p and q 

6qp(T) 

real-time skew between pro- 
cessors p and q when clock for 
processor p equals T 

So 

maximum initial skew 

£ 

maximum clock read error 

^0 

minimum read error, 1 / f c 

P 

clock drift rate with respect to 
real time 

PM 

maximum drift rate expected 
between any two clocks 

Pp 

drift rate of clock p 

Pqp 

drift rate between clocks p 
and q 

E 

maximum clock correction 

Xp 

clock correction calculated by 
processor p 

V 

perceived skew value derived 
from faulty clock reading 


Clock Fundamentals 

The purpose of synchronizing clocks is to pro- 
vide a global time base throughout a distributed 
system. Once this time base exists, transactions 
between members of the distributed system can be 
controlled based on time. For example, the manage- 
ment of redundant data in a real-time fault-tolerant 
computer is simplified if the processors are synchro- 
nized (ref. 9). In the following discussions, the term 
clock refers to a device that provides a time base for 
a processor. A processor thus inherits time-related 
characteristics from its clock. For this reason, we 
sometimes refer to a processor as drifting with re- 
spect to other processors when, in fact, the drift is 
actually a property of the clock. 


A common convention has been that real time is 
denoted by a lowercase letter, as in t or <5, and that 
clock times are capitalized, as in T and A. A clock 
approximates real time with the relationship between 
clock time and real time given by 

t = (l-p)T (1) 

where t is real time, T is clock time, and p is the 

rate of drift of clock time from real time. A clock 

may have some nonzero offset at clock time T — 0, 
as represented by the constant to in equation (2). 

t = {l-p)T + to (2) 


If p is zero, the clock is a perfect clock. If p is 
positive, the clock is a fast clock and accumulates 
time faster than real time. Clocks are considered to 
be digital devices consisting of a crystal oscillator 
and a counter. Ideally, the crystal oscillates at 
frequency f c . Deviations from this specification are 
what cause drift among a set of clocks. The digital 
nature of the counter causes the relation between t 
and T to be discontinuous, as shown in figure 1. The 
error in reading a clock is denoted as e, and for digital 
clocks e has a minimum, £0i °f 1 f fc- Thus, for a 
digital clock the inverse of equation (2) becomes 

T=[(t- <„)/( 1 - p ) j (3) 

where |_ J represents the floor ~ function. 

For a set of clocks, a maximum drift rate p\f is 
chosen so that for any nonfaulty clock p in the set 

M < /W 2 (4) 


The drift between any two clocks p and q in the 
set of nonfaulty clocks is given by 

Pqp — Pq ~ Pp (5) 

with 

\pqp\ < Pm (6) 


2 




The real-time skew b qp that (exists between two 
clocks at some clock time T is given by 

fiqpiT) = t,,(T) - t,,{T) (7) 

Alternatively, the skew can be expressed in terms of 
the difference between two clock values at some real 
time t. The form of equation (7) was chosen, as this 
is the perspective taken in the Lamport and Melliar- 
Srnith proof. 

Synchronizing Clocks 

In the two algorithms considered here, synchro- 
nization is accomplished by periodically executing an 
algorithm that first computes a clock correction value 
and then applies the correction to the local processor 
clock. In order to compute either of the two algo- 
rithms, each processor in the synchronizing set must 
obtain a pm^ived skew A^ between its ( lock and 
each of the other clocks in the set. To obtain A qp , 
processor p must compute the difference between its 
local clock and the remote clock. Processor /; must, in 
effect, read the clock of processor q. Figure' 2 graph- 
ically depicts this process. Bv design, the algorithm 
executes every R time units and takes S time units 
to complete. In the clock subsystem constructed for 
these tests, actual clock values are not transmitted. 
Instead, at predetermined time T s during 5, clock q 
sends a synchronization signal to p. Upon receipt 
of this signal, p reads its local clock and stores this 
value, T,,p\ T qp is then the local clock value for proces- 
sor p taken at a real time corresponding to T s , the 
clock reading for processor q. The perceived skew 
A qp can then be computed as T qp - T s . 



R 


Figure 2. Reading the clock of another processor. 

More precisely stated, the perceived skew values 
are arrived at by the following process: 

1. Each processor broadcasts a synchronizing signal 
at a predetermined time 7*. 

2. Upon receipt of the synchronizing signals from 
other processors, the receiving processor p stores 
its clock value, 'V 

3. The perceived skew is then the stored value, T qp , 
minus I ] s , or 

^<IP — 1 <ip ~ l/t (^) 

Figure 3 represents this process taking place be- 
tween two processors p and (/, with processor q hav- 
ing a clock that is faster than processor p. From the 
graph it can be seen t hat T qp can be t hought of as 
the value of clock p at real time t (J (T s ). or 

T w -~T p (t. ll (T s ))±£, n , (9) 

where T p is the inverse clock function of clock p , and 
e qp is the error inherent in taking T p . 

By using equation (8) with equations (9), (2), and 
(3), the following expression for the perceived skew 
can be derived (see appendix A): 

A qp — —bq p (T s ) i £ + pp L±qp (10) 

An examination of figure 3 will reveal that if q is 
faster than then p qp > 0, h qp > 0, and A r// , < 0. 
To correct its clock, the slower processor p must add 
a positive value to the clock. Since the values of 
A q P will be negative, the resulting correcting value \ 
must be subtracted from clock p (assuming that a 
sign change does not occur in the algorithm). 
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Figure 3. Formulation of T qp . 


Figure 4 graphically depicts the effect of applying 
a correction to a fast clock. The superscripts i and 
z+l refer to synchronization periods, as will be 
discussed in the section “Periods i and i + 1.” In the 
figure, t l refers to the uncorrected clock function and 
P + 1 to the corrected clock function. The correction is 
applied at clock time T c . The following relationship 
exists between the corrected and the uncorrected 
clock functions: 

t i+l (T c ) = t i (T <; ) + (l-p)x i (11) 



Figure 4. Effect of applying correction. 


Some Useful Relations 

The following relations will be used to derive 
the bound formulas. Detailed derivations of these 
relations are given in appendix A. These relations 
hold true provided that a clock correction is not 
applied during the interval from T to ( T + C). 

S qp (T + C) = S qp (T) + Pqp C (12) 

Equation (12) states that the skew between p 
and q at some time T plus a constant C is equivalent 
to the skew that exists at time T plus an amount 
equal to the relative drift rate times the constant. 

fiqpiT) = Pqp ( T ~ T c ) + 6qp(T c ) (13) 

Equation (13) states that the skew between p 
and q during a synchronization period is equivalent to 
the skew at the beginning of the period ( T c ) plus the 
skew accumulated over the period due to the relative 
drift p qp . 

S rq (T) - S rp (T) = 6 pq (T) (14) 

Equation (14) states a relationship that exists 
between the skews of three good clocks, p,q, and r. 

The Proofs 

The statement of the bounding theorem is taken 
largely from references 4 and reference 6. 

Clock Skew Bounding Theorem 

For a set of n processors cooperating in the syn- 
chronization algorithm for all time T through pe- 
riod i, a bound 6 exists on the skew between any 
two of the processors given that at most m of the n 
processors are faulty. Stated mathematically, 

|+T)-4(T)|<6 (15) 

Because this theorem is written in terms of consecu- 
tive periods of time, it is convenient to use proof by 
induction. To do this, we will derive an expression 
for 6 for the first interval, 2 = 0, and then show that 
another expression exists that is true for the following 
intervals. This latter expression depends on charac- 
teristics of the synchronization algorithm, and thus 
separate derivations arc necessary for the ICCSA and 
the Midpoint Algorithm. 

The First Period, i = 0 

At system start-up, assume a maximum skew <5 q 
exists between all good processors in the set. Then, 
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at the end of period 0 with T — R, 
tp(R) - t° q (R) = (1 - Pp)R - (1 - p q )R + < 0 p - to q 
= (pq — Pp)R d" t(\j) ~ q 
— PqpR “f - R)p — ^0q 

\t°(R) - t° q (R)\ <p M R + So<S (16) 

where in expression (16) | t(y p — t() q \ < 6{). Expres- 
sion (16) is thus one constraint on the value of <5, i.e., 
<5 > pmR + «o- 

Periods i and i + 1 

To continue the proof, we will assume that an 
expression for the bound is true for period i and 
show that the same expression is true for i + 1. 
As stated above, this expression will depend on the 
synchronization algorithm. However, we can derive a 
general expression from which the subsequent proofs 
can continue. Refer to figure 5 for a graphical 
representation of the situation that exists between 
periods i and 2 + 1. To reduce clutter in the terms, 
the lack of a superscript will refer to period i and a 
+ superscript will refer to period i + 1 . 


KT); T e t +(T); T € 



t 


Figure 5. Transition from period i to period 7 + 1. 

Using equation (7) for period 2 + 1, we have 

= t+(T r ) - t+(T r ) (17) 

Then using equation (11) to replace the functions 
with t, and then equation (7) again to recombine the 
t functions, we get 

&gp(Tc) = 6qp{T c ) + (Xp - Xq) + PqXq ~ PpXp (18) 

It is assumed that the difference between the p\ 
terms can be ignored. For an error-free system this 
is justified because, wdien considering the worst-case 
skew condition with p q equal to negative /fy, pqXq 
will be of the same sign and approximately equal to 
p p Xp . When clock read errors are present, the worst- 
case read error effect occurs when the error for clock q 
is equal to but opposite to the error for clock p. As 
in the error-free case, the effect is canceled out in 


the px difference terms. In short, when Xp ~ Xq is 
maximized, p q Xq - PpXp ^ minimized. 

Substituting the resulting expression in equa- 
tion (13) written for period i + 1, we obtain 

6+,(T) < 6 q p(T c ) + (x P - X?) + PM R ( 1Q ) 

with R > (T — T c ). Expression (19) will be used in 
the following sections to derive bound expressions for 
the associated algorithms. 

The Interactive Convergence Clock 
Synchronization Algorithm 

The ICCSA is derived for n clocks synchronizing 
in the presence of m faulty clocks. In this algorithm, 
a processor computes the correction by averaging all 
the perceived skew r values A q p. To limit the effect 
of a faulty clock, the A qp are subjected to the test 
that their absolute value be less than some maximum 
expected value A. If A qp exceeds A, A qp is set to 0. 
More precisely 

Xp ~ ~ (2®) 

n , 
q= i 

where A qp = 0 if \A qp \ > A. A value for A is easily 
derived from equation (10): 

A>6 + e+?~A (21) 

Wishing to replace the correction terms in equa- 
tion (19) with an expression based on equation (20), 
we look at the correction terms more closely: 

1 n l u 

Xp ~ Xq = n E Ar '' _ n E Ar '' 
r— 1 r= 1 

u 

= n ~~ 

r= 1 

n—2—m 

= - ~ 
r— 1 

+ — (A pp — Ap q ) + — (Aqp — Aqq) + (\7;> — V?) 

n n n 

The final expression contains four terms, the first of 
which contains values of A qp taken from n — 2 — m 
good processors. The second and third terms have 
readings of the local clock, e.g., A pp . The last term 
holds the readings from the m possible faulty clocks 
(denoted by y)- I ri appendix B, each term is taken 
individually and expanded under assumptions relat- 
ing to those terms and then recombined to obtain 
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Xp - X, < 6qp(T r ) + - {n 1 m) e 

\ n J n 


+ w; (n-m) A+ 2m A 

II 11 


( 22 ) 


Substituting equation (22) into equation (19), we 
get 


*U T ) < 0 w-) + 


2 (n — 1 — m) 


n 


p ( n — rn) . 2 rn 4 

+ — -A + — A + p M R (23) 

n n 


Now wc create an expression for b and assume it holds 
for period i , i.e., that b > 6 qp {T), with T in 2 and with 
b given by 


b > 


2(n - 1 - m) 2m . 

£ + p\[ A H A 


+ 



(24) 


Under this assumption then, by replacing b qp with b 
in (equation (23) we have 




f — \ a _l 2(n - 1 - m) Plain -m) 
n w u ^ 


2 m . 

+ — A + p m R 


(25) 


Now using equation (24) for b , it follows that 


/rm . 2(n - 1 - m) A 2m _ 

&qp{T) < — — — — — e -4- Pm A H A 


n — rn 
n 

n — m 


Pm R — & 


(26) 


which completes the proof. 


The Midpoint Algorithm 

In the Midpoint Algorithm, as suggested by Dolev 
(ref. 8), the correction is computed as the midpoint 
of the span of values of A qp after the rn largest and 
smallest values have been discarded. Stated for the 
case where rn = 1, 

1. Processor obtains all the A qp values. 

2. The A qp arc ordered so that A miri < A mill /... < 

^max' — ^max- 


3. Discard A m j n and A max and use the new mini- 
mum and maximum, A min / and A max ,, to com- 
pute the correction as 

,, __ ^min' + ^max' 

A p ~ 2 

This algorithm has the property that the clock read- 
ing of a faulty processor will not be used to com- 
pute the correction unless it is bounded by good clock 
readings. This results in it being possible to derive a 
tighter bound. 

In the following sections, an expression for \ p is 
derived by first considering the case with no errors, 
then with some clock read error £, and finally with 
an arbitrary faulty clock reading. 

The ideal ca$e. In the absence of a faulty clock 
and read errors, all good processors in a synchroniz- 
ing set will place the processor readings in the same 
order. Take, for example, the four-processor system 
(p, q, r, .s) where 

tp(T) < tq(T) < tr(T) < ts(T) 

Then, for any member i in (p. q, i\ s) 

A p i < A (p < A r j < A s j 

All good processors will then use clock readings from 
the same two processors to compute their respective 
corrections. (In the above example, this would be 
A (jj and A r/ .) This is equivalent to the processors 
using a single clock reading which is at the midpoint 
of these two clock readings (Anid (?*)). Thus using 
equation (10) with £ = 0, we have 

Xp — Amid,/; = — ^mid(^s) T PpA n ,id.y> (28) 


Including read error . Any read error present 
in the clock readings will affect the clock correction 
by at most the read error £: 


Xp 


^ - T ^niax' ^ ■ 


2 


±e 


^mid./y i E 

~ ^niid,p(7s) + PpA m ’id,p i £ 


(29) 


Including a faulty clock . In reference to fig- 
ure 6, consider that the maximum and minimum 
readings taken from good clocks differ by at most 
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6 + 2e. The algorithm guarantees that if a faulty 
clock reading is used in computing the correction, 
it is bounded by good clock readings. Thus, the 
maximum error that a faulty clock could cause is 

1/2(6 + 2 e). The expression for the correction includ- 
ing both read error and error due to a faulty clock 
reading becomes 

Min' ± (Mr^ + £ ) + A niax' ± (Mr^ + 0 

Xp = 2 

_ ^\nin' ^tnax' ± 

= A niid.p ± + e ) 

Xp = — ^inid,p(r!s) + /tyMuid.p A ^4 £ ) 

A maximum correction E can be obtained by using 6 
and A for the maximum values of 6 mi( p p and A mi( p p , 
giving 

r>« + £*iA±(j+ E ) 

>“ +£ + 5|1a (3i) 

4 Z 

Now using equation (30) in equation (19), we obtain 

< b qp {Tc) + [<‘w,( 7 Y) - gr,)| 

+ (p 9 A mi ,i, (/ - ^A mi ,i. p ) + 2 (- + e) + pR (32) 


Substituting equation (35) for ti in equation (34), we 
get 

«+ (T) < 4e + 2p M A + 2p\j R < t) (36) 
which completes the proof. 


.El 


m i n 

, e | e , 




^max 

ei c. 


J max 
£ I £ 


<6 + 


Figure C). Set of perceived skews taken from good clocks. 

If the effect of faulty processors were to be ignored 
(rn — 0), then equation (35) becomes 

6+,(T) <2e + pm A + pm R (37) 

and the clock bound is 

6 > 2€ + pm& + pmR (38) 


Experimental Verification 

To experimentally verify the derived skew bounds, 
several tests were performed in which the effect of 
varying one parameter of the skew bound expression 
was measured while the remaining parameters were 
either held constant or zero. The parameters are 
6 0 , e, />, rn, n, and R. For the clock subsystem that 
was actually tested, the number of clocks n was kept 
constant at four, and thus m was limited to (0,1) for 
both algorithms. It was decided that if p is tested, 
it is not necessary to test the effect of varying the 
synchronization period R. The following test cases 
were then generated: 


Ignoring the difference between the pA terms (as 
was done in cq. (18) with the px terms) and using 
equation (14) on [fi ni(} (T s ) — S mp {T s )\. we get 

6+(T) < 6qp(T r ) - M^s) + 2 ( b - + e) + PM R (33) 

We then use equation (12) with T r - T s + A to obtain 


1. 6 — 0 with 6o = 0, rn = 0, £ = 0. and p - 0 

2. 6 — /(6o) during the first period with p = 0 

3. 6 = /(6 o) during the first period with p = C 

4. 6 = f(e) with m = (0, - t * = 0, and 6 q — 0 

5. 6 — f(s) with rn — (0, 1), p = C. and 6<) = 0 

6. 6 = f(p) with m = (0, 1),£ — 0, and 6o = 0 

7. 6 = f(p) with m = (0, 1),£ = C , and 6p = 0 


t>qp(T) < 6qp(T, ) - t>, lp (T r ) + PA/ A + 2 (^ + cj + pa/ R 

6+,{T) < PM A 4- 2 ( Q + PmR (34) 

Now to continue the induction, we assume the fol- 
lowing expression to be true for period i : 

6 > 4s -h 2p\jA + 2p,\/7? (35) 


In all the tests, the read error is treated as a 
random variable with a mean of zero. This is not 
the case in most communication systems. However, 
the expected value of the communication delay is 
often knowrn and can be subtracted from the clock 
readings in the synchronization algorithm, so that 
the resulting effect is a read error with zero mean. 

In addition to functioning as a synchronizing cir- 
cuit, the clock subsystem must be able to support 
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the test plan. The following capabilities were then 
designed into the clock subsystem and the experi- 
ment support environment: 

1. Ability to sustain long-duration data acquisi- 
tion of internal variables without perturbing 
the system function 

2. Availability of a global clock that can be read 
by each processor under test; the global clock 
will represent real time 

3. Ability to set the starting skew 6q of each clock 

4. Ability to set the drift rate of each clock with 
respect to real time, i.e., the global clock 

5. Ability to set the read error of each clock 

6. Ability to emulate a faulty clock, especially a 
malicious liar 

The following sections describe the clock sub- 
system and experiment environment. 

Design of Clock Subsystem 

The clock subsystem is designed as a synchro- 
nization peripheral. This primary function is then 
augmented to provide the data acquisition and con- 
trol necessary to accomplish the tests proposed in 
the previous section. The next section will describe 
the design of the primary synchronization function. 
This is followed by a section on the actual design, 
which includes the test augmentations. In these and 
the subsequent sections, the term clock tick is used 
to refer to one increment of digital time. Practically 
all the parameters are stated in terms of clock ticks 
instead oi time. A clock tick is easily converted to 
time once the base frequency of the clock is known. 

A clock synchronization peripheral. As men- 
tioned previously, the ICCSA was first used in the 
SIFT computer. This implementation was tested 
(ref. 10), and it was found that the clock skews 
were due primarily to large clock read errors. It was 
proposed then that a simple hardware enhancement 
could greatly reduce the read error, tighten the clock 
synchronization, and thus increase the efficiency of 
interprocessor communication. While it is possible 
to put the entire clock function in hardware, for the 
purposes of this test it is convenient to have the al- 
gorithm in software so that alternate algorithms can 
be tested. Having the algorithm in software also en- 
hances data acquisition and fault simulation. 

Figure 7 is a block diagram of how the clock 
functions are distributed between the clock periph- 
eral hardware and the synchronization software. The 


clock hardware monitors a communication channel 
for the presence of a synchronizing signal. When a 
sync signal is detected, the hardware latches the local 
clock value and stores it in a register related to the 
processor that sent the signal. The clock hardware 
also generates a sync signal at a specified time T s 
and places the signal on the communication channel. 
These functions are done most efficiently (i.e., the 
lowest read error is realized) if they are integrated 
with the communications and networking protocols. 
The clock peripheral also generates an interrupt to 
the host processor to indicate the end of the pe- 
riod. The processor then executes the clock algo- 
rithm, reading the clock read registers, computing 
the correction, and correcting the clock. 



Figure 7. Block diagram of clock functions. 


Several considerations must be made to properly 
design the clock peripheral. The ICCSA requires that 
all clock readings greater then A be ignored. This is 
equivalent to a buffer of size A existing before and 
after the synchronization time T s (sec fig. 8). The 
clock hardware can easily be designed to enforce the 
rejection of signals received outside this window by 
clearing all clock read registers at the beginning of 
the window and inhibiting the update of the regist ers 
at the end of the window (when the interrupt to the 
processor is generated). 

^ R 


I ^ 


s 


START T s END 

A . s A 


Figure 8. Synchronization window. 


Thought must also be given to the clock itself. 
The clock must be corrected. While at first this may 
sound trivial, several factors should be considered. 
A read error equivalent to 1 / f c could be induced 
every time a clock is read or written. Thus, by 
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reading the clock, adding the correction, and writing 
the new value, two clock ticks of read error can be 
accumulated. Also, since it takes the processor a 
finite amount of time to perform the correction, it 
is possible that additional ticks will be lost during 
the correction. Correcting the clock by adding the 
correction is undesirable because clock time will be 
either “lost” or repeated, and then care must be 
taken not to “skip over” or “reschedule” an event. 
Alternative correction methods can be designed that 
add pulses to or delete pulses from a clock oscillator 
input, as necessary. As will be seen, this is the 
method used to adjust the drift rate between the 
processors. To avoid possible interaction between 
the application of the correction and the drift rate 
setting, another correction method was developed. 

In the clock circuit tested, the correction is ap- 
plied by moving the synchronization window (which 
defines the end of the frame). Normally this would 
result in larger skews because the clocks will drift for 
an additional frame before the correction takes effect. 
This is indeed what would happen. However, during 
this test no other tasks are scheduled off the clock 
during the frame. Thus, moving the synchronization 
window is a way of applying the correction for the 
purpose of this test. Measurements are not affected 
because data are only taken during the execution of 
the synchronization algorithm, and by this time, the 
correction for the last frame has already been ap- 
plied. An additional benefit of using this method 
is that the length of time taken to compute the al- 
gorithm (including any interrupt latency) does not 
affect the experiment. This allowed a great deal of 
freedom in coding different algorithms, fault models, 
and data acquisition. 


clock frequency. If we let p be defined as 

p= f -^ (fr < It) (40) 

Jr 

then equation (39) can be written as 

V = 1 ~ — (0.0 < p < 0.5) (41) 

P 

For a drift rate of 10 - ^, v = 99998. 

Read errors and faulty clock behavior can be 
programmed by varying the sync strobe time. To 
present different errors to each of the remote clocks 
(a form of malicious behavior), a SYNC pulse must 
be independently generated for each remote clock. 
Thus, three SYNC register /comparators were used 
in the final circuit design. 

Figure 9 is a block diagram of the clock synchro- 
nization peripheral. The circuit is designed for four 
clocks (one local and three remote) and assumes a 
dedicated connection to the remote clocks. An oscil- 
lator drives a counter of sufficient length to resolve 
a frame. Five register /comparator blocks define the 
START window time, SYNC times, and END win- 
dow time (T c ). The START strobe clears and en- 
ables the STORE n registers. The SYNC strobes are 
broadcast to the remote clocks. The END strobe dis- 
ables the STORE n registers, interrupts the proces- 
sor, and clears the clock (counter), beginning a new 
frame. Three remote clock strobes are gated through 
the enable circuitry to the STORE n registers. 
On receipt of a synchronization strobe, the current 
clock value is latched into the associated STORE n 
register. 


Test augmentation . The clock peripheral de- 
sign is augmented to allow the adjustment of the 
oscillator drift rate, the setting of read error, and 
the simulation of a malicious liar. To adjust the drift 
rate, the oscillator input of the clock counter is driven 
by a pulse deletion circuit. The pulse deletion circuit 
has as input a reference oscillator signal (the global 
clock oscillator) and a 16-bit unsigned integer value. 
The circuit loads the 16- bit value in a down-counter 
and deletes a pulse from the reference oscillator sig- 
nal on overflow. A value of 0 will cause every other 
pulse to be deleted; a value of 1 will delete every 
third pulse, and so on, so that the clock frequency is 
defined as 


where v is the 16-bit value and f r is the reference 


Experiment Environment 

The clock peripherals were installed on an exist- 
ing fault- tolerant processor (FTP) test-bed (ref. 11). 
The FTP is hosted from a VAX computer through 
a dual port memory. In addition, each channel of 
the quad FTP has an additional dual port memory 
channel to separate VAX computers. These channels 
were dedicated to data acquisition. A sixth VAX 
computer with a windowing interface was used to 
control the experiment. The FTP is a tightly cou- 
pled computer. Initial skew is then easily controlled 
from the base skew of 6q = 0 provided by the FTP. 
The synchronization algorithm is loaded into FTP 
RAM and configured for the test trial. The FTP op- 
erating system is then started from ROM. After the 
FTP stabilizes, control is passed to the synchroniza- 
tion algorithm and the FTP clock synchronization is 
disabled. 
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Figure 9. Detail of clock synchronization peripheral. 


Another component of the experiment environ- 
ment is the global clock. The global clock has a base 
frequency of 2 MHz and a resolution of 32 bits. The 
output of the global clock can be read by each chan- 
nel and is assumed to be real time. To establish the 
global clock as real time, its 2-MHz base frequency is 
fed to the clock synchronization peripherals as the 
reference frequency. Thus, in the absence of any 
programmed drift rate, the clock synchronization pe- 
ripherals are perfectly synchronized. 

Results 

Several tests were run to verify the functionality 
of the system. The following runs were made with 
the synchronization algorithm disabled: 

1. (m = O,/0 = O,£ = O, and £q = 0) to test the 
global clock 

2. (to = 0, p > 0, e = 0, and = 0) to test drift 
rate circuits 

3. [rn — 0,p = 0, e = 0, and > 0) to test 
setting initial skew 

4. (to = 0,p = (),£ > 0, and 6 () = 0) to test 
setting the read error 


With the synchronization algorithm enabled, sev- 
eral tests were run with <5 0 > 0 and p > 0, and it was 
found that equation (16), the i = 0 synchronization 
constraint, held. The next several sections present 
the results of testing the ICCSA and the Midpoint 
Algorithm. 

The ICCSA . In reference 6, six constraints are 
listed that must be met if the bounding theorem is to 
hold for a clock synchronization system executing the 
ICCSA (see table I). These constraints include the 
skew bounds (C5 and C6), the maximum perceived 
clock skew A (C4), the maximum clock correction £ 
(C3), the minimum time allocated to the synchro- 
nization process S (C2), and the minimum length of 
the synchronization frame R (Cl). A synchroniza- 
tion subsystem based on these constraints must have 
the property that a processor can read a remote clock 
at. a time when the remote processor is not execut- 
ing the synchronization process. That is, the remote 
clock must be accessible for external reads outside 
the scope of its own synchronization process. This is 
clearly not the case with the design used in this te st. 

Because a remote clock is read with the coopera- 
tion of the remote synchronization process, the syn- 
chronization windows must allocate adequate time 
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before and after the synchronization time T s in order 
to be sure of capturing all good clocks. This time is 
at least 6 + e. In these tests, the window was set at 
2 times the maximum perceived clock skew A, with 
the synchronization time T s in the center of the win- 
dow (see fig. 8). Thus, the period R is determined 
by the END window register. The START window 
register is set to END — 2 A and the SYNC registers 
are set to END — A. 


Table I. Constraints for Old Theory ICCSA 


Constraint definition 

Constraint, relation 

Cl: minimum period time 

R > :is 

C2: minimum algorithm time 

s > >: 

C3: maximum correction 

E > A 

C 4: maximum perceived skew 

A > A + e + 

C5: maximum skew 

^ > fy) + PM R 

Ci'y. maximum skew 

h > 2e -1- p\f{2S + A) + f) w A 


+ pm <» + >:) 


Table II. Constraints for New Theory ICCSA 


Constraint definit ion 

Constraint relation 

Cl: minimum period time 

n > s + >: 

C2: minimum algorithm time 

S > 2A 

C3: maximum correction 

K 

IV 

t> 

C4: maximum perceived skew 

A >t> +f + ^A 

CT>: nuiximurn skew 

^ > <^() + PM R 

C6: maximum skew 

+ A + („- m )ruR 


The constraints as defined for these tests are listed 
in table II. The only expression that remains equiv- 
alent to table I is C5. The difference in C4 may be 
due to the difference in S as described in the previ- 
ous paragraph; C3 defines the maximum correction 
possible if all n — 1 clocks return a difference of A; 
C2 comes directly from the above discussion. Fi- 
nally, R must be at least as big as 5, with room for a 
correction. 


Figure 10 shows, for one series of tests, the bound 
for the old theory (table I, C6), the bound as derived 
in this paper (table II, C6), and the actual data. 
These plots are of maximum clock skew (in ticks) 
versus drift rate. The data were taken at large drift 
rates with a constant read error of 200. Figure 10(a) 
displays zero- fault-tolerant performance (m = 0), 
and figure 10(b), single-fault-tolerant performance 
(m =1). The bound as derived in this paper exactly 
predicts the performance of the result. 


x 1() 3 




(1)) m ~ 1. 

Figure 10. ICCSA test, results. 


The Midpoint Algorithm . A theory based 
on the Midpoint Algorithm was derived in refer- 
ence 7 and interpreted in reference 2. Table III lists 
the constraints for the old theory in terms of the 
symbols used in this paper (see appendix C). Ta- 
ble IV contains the constraints for the theory for the 
Midpoint Algorithm as derived in this paper. The 
synchronization process was identical to the ICCSA 
with the exception that the Midpoint Algorithm was 
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executed at T c . Figure 11 plots the clock skew bound 
predicted by the old theory, the theory derived in this 
paper, and the actual measured results versus drift 
rate. As can be seen, the measured clock skew is 
well below that predicted by the new theory. This 
is not due to an inaccuracy in the theory, but to an 
inability to replicate worst-case conditions with the 
clock subsystem. This phenomenon will be explained 
in more detail in the section Simulating a Malicious 
Liar. 

Case Studies 

The parameters used in the verification tests are 
obviously far worse than can be expected in an 
actual system. However, now that the theory has 
been verified under these extreme conditions, it is 
reasonable to ask what level of performance can 
be expected under nominal conditions. The case 


studies listed in table V were generated to probe this 
area. The case studies deal primarily with read error 
and synchronization period, as these are the most 
significant contributors to the clock skew. 

A read error occurs every time a digital clock is 
read. It is believed that the minimum read error that 
will be obtainable in most synchronization systems is 
1 tick. This tick of read error is added when, as is 
the case with the subject clock subsystem, the local 
clock is read in response to the strobe generated by 
the remote clock. In this case the remote clock is 
not actually read, but generates an event signal that, 
by definition, occurs at clock time T s and, therefore, 
does not include an error component. A similar 
situation would exist if the remote clock were to be 
read in response to a request from the local clock 
(given that there were no other overhead). Case 1 
covers this best-case situation. 


Table III. Constraints for Old Theory — Midpoint Algorithm 


Constraint definition 

Constraint relation 

Cl: minimum period time 

R > 3A + ?f-6 0 

Cla; required lower bound on t 

+ 2p M {h& + tfo) + 2 P 2 M f>U 

+ 1 

C2: minimum algorithm time 

S > A 

C3: maximum correction 

S> | + A 

C4: maximum perceived skew 

A > 6 4- e + ^-A 

C5: maximum skew 

Assume C6 dominates 

C6: maximum skew 

4f ^1 - + 2 PAf(2A + <§o + R) + f? M b o 

p'm + 1 


Table IV. Constraints for New Theory Midpoint Algorithm 


Constraint definition 

Constraint relation 

Cl: minimum period time 

R> S + £ 

C2: minimum algorithm time 

5 > A 

C3: maximum correction 

M 

IV 

+ 

t> 

C4: maximum perceived skew 

A > <5 + e + ty-s 

C5: maximum skew 

6 > f>0 + PM** 

C6: maximum skew 

6 > 4e + 2pm A + 2 pfrfR 
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Figure 11. Midpoint Algorithm test results with m = 1. 


Table V. Case Study Parameters 


Case 

Drift, rate, p 

Period, /f, 
ticks 

pR, 

ticks/ period 

Read error, e, 
ticks 

la 

1.00 x HP 5 

1.00 x 10 4 

0.1 

1 

lb 



1.00 x 10 5 

1.0 

1 

lc 



1.00 x 10 5 

1.0 

1 

2a 



4.00 x 10 4 

.4 

4 

2b 



1.00 x 10 5 

1.0 

4 

2c 



4.00 x 10 5 

4.0 

4 

3a 



1.00 x 10 5 

1.0 

10 

3b 



1.00 x 10 5 

1.0 

10 

3c 



1.00 x 10 e 

10.0 

10 


If both the local clock and the remote clock are 
read in response to asynchronous events generated by 
the processor, then 2 ticks of error would be added to 
a clock read. Similarly, 2 ticks of read error can also 
be added when a clock is corrected. This is again 
due primarily to the asynchronous nature of clock 
reads and writes. If the clock correction circuitry 
is designed properly, this error will not be incurred. 
Case 2 covers the situation when the read error e 
is 4, with 2 ticks added during clock read and clock 
correction. 

To include a somewhat less than optimal situa- 
tion, the read error is set to 10 in case 3. 

Each case consists of three subcases where the 
drift rate is set so that the accumulated drift over one 
period is equal to Via of the read error in subcase “a,” 
1.0 tick in subcase “b,” and the entire read error in 
subcase “c.” This leads to two redundant cases (lc 
and 3b). 


Figure 12(a) is a plot of all three cases. Fig- 
ure 12(b) plots cases 1 and 2, which represent best- 
case conditions. Data are plotted for both the ICCSA 
(dashed lines) and the Midpoint Algorithm (solid 
lines) and for both zero- fault- tolerant (filled symbols) 
and single-fault-tolerant cases (empty symbols). 




(b) Cases 1 and 2. 
Figure 12. Case study results. 


Discussion of Results 

The results of this study span a broad spectrum 
of subject matter including clock algorithm perfor- 
mance, design methodology, and techniques of worst- 
case testing. The following sections address these 
issues. 

Clock Algorithm Performance 

As can be seen from comparing the fault-free and 
single-fault cases in figures 12(a) and 12(b), a per- 
formance penalty of 100 percent is paid to protect 
the system from faults. It is interesting to note 
that this penalty is the same for both algorithms. 
If a clock skew dead band is made part of every 
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communications exchange, then designers must con- 
sider whether they are willing to pay this penalty 
to protect the system from a rare form of malicious 
behavior. 

The equations for the clock skew upper bound 
suggest that the component of clock skew due to ac- 
tual drift (pR) can be reduced to an insignificant level 
if R is made small enough. This is not thought to be 
possible, since, in the absence of read error, no cor- 
rection will be made for a series of intervals until a 
significant skew has accumulated. A correction will 
then be made. This was in fact observed indirectly. 
Direct observation was not possible because our sys- 
tem had 1 tick of read error, minimum. 

The indirect observation was made by first taking 
one data set with zero additional read error and zero 
drift rate. What is observed is the minimum read 
error of the system. This was done for several thou- 
sand dock readings, with none exceeding ±1 tick. To 
observe the effect of pR < 1, the same system was 
then run with pR = 0.1. Within this series, occa- 
sional readings of ±2 were observed, thus supporting 
the conjecture that the pR term actually contributes 
an amount equivalent to the function ceiling(pR). 

The Midpoint Algorithm outperforms the ICCSA 
and is the clear choice. Remembering that the 
“a” series subcases are hypothetical with pR < 1, 
the next best design is case lb (e — 1 ,pR. — 1), 
which yields a single-fault-tolerant skew bound of 
6 ticks. While this kind of performance is possible 
over dedicated links, it may not be possible to design 
a general-purpose communication protocol that can 
support both efficient transfer of normal traffic and 
very low read error. 

If it is necessary to allow for greater read error, 
as represented by case 3, the designer has a wider 
choice in selecting the synchronization period. In 
this case, the use of a minimum synchronization 
period (i.e., with pR = 1) may yield only marginally 
tighter clock skew's because the read error dominates. 
The frequent synchronizations may produce more 
overhead on the communications channel than is 
saved by virtue of the resultant tighter clock skews. 

Design Methodology 

One of the areas in which clock synchronization 
is used is highly reliable fault-tolerant architectures 
such as those in military and commercial aircraft. 
The high reliability requirements put on these de- 
signs (probability of failure = 10 -9 per mission) pre- 
clude testing as a means of validating that this re- 
quirement has been met. One of the methods that 


has been suggested for this purpose is formal verifica- 
tion. A formal verification methodology would entail 
the use of a specification language and the construc- 
tion of a hierarchical theory written in that language 
that could be proven to show that the final design 
meets the highest level specification. Automated the- 
orem provers are often used to facilitate this task. 
A good example of this method is HDM (ref. 12) 
as used on SIFT. Most recently this has matured to 
EHDM, which was used by Rusliby (ref. 6) to rederive 
the clock theory originally invented by Lamport and 
Melliar-Smith. In reference 6, Rushby reports that 
the rigor enforced by the use of the theorem provers 
led to the uncovering of several inconsistencies in the 
original, hand-derived theory. 

The purpose of experimental verification as re- 
ported in this paper was to demonstrate that the 
formal theory was indeed correct. What w^as found 
was that although the theory w'as correct in that it 
predicted a bound that was never violated, the bound 
was only a bound and not a model for the actual cir- 
cuit performance. With the insight gained by experi- 
mentally observing the behavior of the circuit, it w 7 as 
possible to derive a more accurate theory. Thus, al- 
though testing cannot be relied upon to verify highly 
reliable components, it becomes an integral part of 
deriving the theory, which can then be used to predict 
the performance of the circuit into the unobservable 
regions. While this may sound obvious to those w r ho 
have practiced such techniques, it has been observed 
that individuals tend to be heavily biased toward ei- 
ther the “design and debug" or “theorize and prove" 
camps. 

Figure 13 is an attempt to illustrate an op- 
timal design methodology. The two axes delin- 
eate time spent testing and theorizing. A vector 
DMV is drawn whose length represents design op- 
timality. It is proposed that the optimality is di- 
rectly proportional to the correctness of a design 
and inversely proportional to its cost. The locus 
of points traced by this vector suggests that if too 
much emphasis is placed on either testing or the- 
ory, design optimality suffers and that the opti- 
mum design is reached by applying those techniques 
best suited for the particular problem. As demon- 
strated in this work, verification of predicted val- 
ues of physical quantities is well suited to testing. 
Testing will also provide behavioral insight, which 
aids in the construction of provable and realizable 
theory. As will be seen in the next section, test- 
ing cannot be relied upon to quantify worst-case 
behavior. 
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Figure 11. Anticipated malicious liar behavior. 


Simulating a Malicious Liar 

To experimentally verify the clock theory, special 
circuits were added to the clock peripheral circuitry 
to enable the simulation of malicious faults (see 
the section "Test augmentation”). During testing 
of the ICCSA, the worst-case behavior of a lying 
clock was more difficult to simulate than originally 
anticipated, and the special circuitry could not be 
used to simulate worst-case conditions without great 
difficulty. Moreover, for the Midpoint Algorithm, 
worst-case conditions could not. be simulated at all. 

Figure 14 shows the faulty behavior that was 
assumed during the design of the test equipment. 
The figure illustrates the time line of three processors 
p, q , and r, with p and r being good processors and q 
being a lying processor. If p is a slow processor with 
respect to r, then q would send a synchronization 
signal to p just prior to t he end of the synchronization 
window to give p the perception that it was a good 
deal faster than q and thus cause p to apply a 


correction that would slow its clock even further. 
Conversely, q would signal r at the beginning of the 
window and cause r to apply a correction that would 
speed up its clock. 

In practice, the difficulty with doing this is that 
although it is possible to anticipate the beginning 
and ending window times for r and p with respect 
to q for the first frame, it was observed that worst- 
case skew is not obtained until several frames later. 
This behavior is illustrated in figure 15. Consider 
the case in which processor p uses the ICCSA. Pro- 
cessor p will read a clock difference of A from q in 
frame 1. Processor p uses this value as part of the av- 
eraging process to compute the correction. The cor- 
rection computed by processor p will thus have an 
error of A/4 (for four processors). Processor r, on 
the other hand, will apply a correction with an equal 
but opposite error with the result that the synchro- 
nization windows of p and r have been driven A/2 
farther apart. Thus, for q to again send worst-case 
synchronization signals, it must now take this addi- 
tional skew into account, as illustrated by the second 
frame in the figure. The correction error would then 
become (A + A/4)/4. The correction is then increas- 
ing by amount A/4^, where k is the frame number. 
The skew between p and r would increase until the 
additional error becomes insignificant, i.e., A < 4*\ 
This typically took five frames when large drift rates 
made large synchronization windows necessary. 



It was decided, after having observed this behav- 
ior, to model the malicious behavior from the per- 
spective of the good processors instead of creating 
the erroneous signal on the faulty processor. This 
was done by providing the synchronization algorithm 
with a parameter that indicated which remote clock 
was to be considered a liar and in which direction 
it was lying. The good processor then substituted 
its START or END window value for the actual 
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reading of the faulty clock, thus simulating the ef- 
fect described above. 

Worst-case conditions could not be simulated 
with the Midpoint Algorithm because of the lack of 
sufficient processors to create the necessary condi- 
tions. Worst-case conditions are a combination of 
maximum drift, maximum read error, and the pres- 
ence of a malicious liar. In the Midpoint Algorithm, 
the two outlying clock differences are discarded and 
the remaining two averaged (for four processors). 
When a malicious liar is present and behaves as de- 
scribed above, it will cause the fastest and slowest 
clocks to include their clock difference readings (0) 
in the correction computation. Normally, the fastest 
and slowest clocks would be at the extremes and not 
be used. The “self’ clock readings do not contain 
any read error, so that the worst-case skew is not 
achieved. In a system of five or more clocks, it would 
have been possible to arrange the parameters to cre- 
ate worst-case conditions. 

In conclusion, testing cannot be relied upon to 
create worst-case behavior. The complex interactions 
often confound cursory analysis; the result is that 
something other than worst case may be observed, 
with the danger then that the system will be designed 
around these misleading specifications. Developing a 


theory that predicts worst case provides a checking 
mechanism that when the theory prediction does 
not match the observation, immediately raises the 
question of which is at fault. For a highly reliable 
design, these kinds of discrepancies must be known 
and resolved. 

Concluding Remarks 

New theory has been developed and experimen- 
tally verified for the Interactive Convergence Clock 
Synchronization Algorithm and the Midpoint Algo- 
rithm. The Midpoint Algorithm is capable of achiev- 
ing tighter synchronization than the Interactive Con- 
vergence Clock Synchronization Algorithm. Both 
algorithms suffer a 100-percent penalty to protect 
against one fault. The new theory outperforms exist- 
ing theory that was developed without the benefit of 
the insight gained during experimental verification. 
However, it is not adequate to rely on testing pro- 
cedures to uncover worst-case behavior. Testing and 
theory go hand in hand to produce optimal designs. 
This is especially true for highly reliable systems. 

NASA Langley Research Center 
Hampton, VA 23665-5225 
May 5, 1992 
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Appendix A 

Proving Equations (10), (12), (13), 
and (14) 

Proving Equation (10) 

To prove equation (10), that is, 

Aqp = —6qp(T s ) ±s + p p A qp (10) 

we start with the definition of A qp , equation (8), 
which is 

A 9P = Tq P - T s (8) 

and using equation (9) to expand T qp and expressing 
T s as the value of clock p at a real time when clock p 
reads T s , we get 

A qp = T p (t q (T s )) - T p (t p (T s )) 

Using equation (3) to expand the clock functions T p 
and realizing that the second term incurs no read 
error, we have 


tq(T s ) ±£ — ta p 


a. 

0 

1 

a. 

-Ki 

1 - Pp 


1 PP 


Finally, combining terms and using equation (7), we 
get 


we start with equation (7) and substitute equa- 
tion (1) as follows: 

t>qp(T + C)= tp(T + C)~ tq(T + C ) 

= [(1 - P P ){T + C) + t 0p ] 

~ [(1 - Pq){T + C) + 

= 3qp{T) + ( p q ~~ Pp)C 

= &qp{T) + PqpC 

Proving Equation (13) 

To prove equation (13), that is, 

6 qp (T) = Pqp (T - T e ) + 6 qp (T r ) (13) 

we rearrange equation (12) and substitute C = 
— (T — T c ). 

Proving Equation (14) 

To prove equation (14), that is, 

6 rq (T) - 6rp(T) = 6 pq (T) (14) 


A qp = 


— fiqpiTs) ^ £ 

1 “ Pp 

&qp ( Ts ) i £ T PpAqp 


we use equation (7) and write 

6rq(T) - 6rp(T) = [t q (T) - t r (T ) ] - [l p ( T) - t r (T)] 


Proving Equation (12) 

To prove equation (12), that is, 

S qp (T+C) =6qp{T)+ Pqp C (12) 


= [t q (T) - t p (T)} 

= S Pq (T) = -6 qP (T) 
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Appendix B 


The Local Processors 


The Expansion of \ p - \ q 

We expand the term \ p - \ q : 


^ n-2- m 

~ Xq — “ (A, 7j - 


"b ^ (App Apif ) 


+ “ ( A '/;> - A w) + — (V,, - V?) ( B 1 ) 


This expression contains four terms that can be 
considered in three groups. The first term represents 
the ( n - 2 - m) good processors. The second and 
third terms represent good processors, one of which 
is a local clock. The third term represents readings 
taken from bad processors. 


Taking the two terms that include local processor 
readings, we write 


^/»/) + (^v/i A/,/ J 


~ n i 1 — - {-(>, „,(T S ) ± 5 + p 9 Ap,]} 

+ - {[~MTd ± E + f>,A,,p} - M W (T,)]} 


+ n il it|- [— ^ W (T,)]} 


+ CL& _ Cl a 

n "" n « 


Finally, ignoring the difference between the pA terms 
we obtain 


The Good Processors 

We will first reduce the term A rp - A rq using 
equation (10). 

Arp — A V q = [ — 6 rp (T s ) i £ + PpA r p] 

~~ [~firq{T s ) ± € + />gA r g] 

— < [8 rq (T s ) — ^rp(T s )] 2^ 

“k {Pp^rp — PqA rq ) 

— [<WTs) ~ “f 2£ (B2) 

Here, as before, the difference between the pA values 
is ignored. These results are replaced in the sum (B1 ) 
and simplified as follows: 


1 

n 


1 

n 


n -2- m n-2- m 

E (A r; , - A r? ) < - 

r= 1 ' r- i 

x { [<Er/(T< ) — <V/>(Ts)J + 2.5:} 

n -2 - vi. n—2- rn 

E (A,.„-A,,)<I £ 


r=l 


r=1 
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X (MT) - t> rp (T s )} + £ (B3) 


^ (Aqp A w ) 

= ~ {[-v(T)i - [-V/(T,m 
+ ,7 { [-fiw.fr.)] - I-fi w (r.)] } + -e (B4) 

n n 


Good Plus Local Processors 

Combining equations (B3) and (B4) by replacing 
the first two terms on the right-hand side of equa- 
tion (B4) in the summation of equation (B3), we get 


Good + Local 


I n—m 

- E M^) - MTs)] 

V— i 


+ 


2 ( 7 / — 1 — 777 

£ 

n 


(B5) 


Taking a closer look at the expression within the 
summation we have, with T r = T s + A, 

M r «) - b rp (T,) = fi rq (T c -A)- 6 rp (T r - A) 

Now using equations (12) and (5), we get 

firry (Ts) ~ 6 r p(Tfi) — t) rq (T r ) - b,p(T r ) + p qp A 

Finally, application of equation (14) yields 

^rq(T.s) ~ d,p ( T s ) = —bqp(Tf ) + Pi/pA (B6) 



Now, substituting equation (B6) into equation (B5), 


Good Plus Local Plus Bad Processors 


Good + Local 


( rn — n 
n 


&qp{Tc) + 


2 (n - 1 - m) 


p A /(n-m) A 


(B7) 


The Bad Processors 


Recalling from the ICCSA that all perceived 
skews are limited to a maximum of A, we have 


Using equations (B7) and (B8) in the original 
expression gives 


- Yn < 


(r^) M r c) + 2(n - 1 - m 4 


Xp Xq — v „ I ^ ' - n 

2rn 


+ 


-A + — A 
n n 


m 

n 


(Vp 


, 2m 

Vq) < A 

H n 


(B8) 
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